Method and device for the creation of pseudo-holographic images

ABSTRACT

A method and a device enable the creation of three-dimensional images with more than two perspectives (e.g. pseudo-holographic images), especially to be reproduced with the aid of an autostereoscopic display or an autostereoscopic screen, from fed images having, in particular, only two perspectives, e.g., a left and a right image channel. Also, a related device creates and reproduces three-dimensional images having more than two perspectives, especially in the form of an autostereoscopic multi-user visualization system.

The invention is related to a method and a device for the creation ofthree dimensional images with more than two perspectives (such aspseudo-holographic images), especially for displaying on anautostereoscopic display or an autostereoscopic screen from fed imageswith, in particular, only two perspectives as, for example, from leftand right image channels. The invention is also related to a device forthe creation and display of three dimensional images with more than twoperspectives, especially with an autostereoscopic multiuservisualization system (multiview system).

Autostereoscopic visualization systems will allow one or more viewerslooking at an autostereoscopic display or an autostereocopic screen, toview a three dimensional image without visual aids such as red/blueglasses, shutter or polarization glasses. For this purpose for exampleparallax barrier systems or lenticular systems, which will be attachedto a display, are used. But because, as described in FIG. 1, one or moreviewers B1, . . . Bn will be at different angles relative to theperpendicular direction of the display or the screen DA, there must bemore than two perspectives generated and presented to the left or righteye in order to allow in all positions S1 to Sn and all viewers B1 to Bnrespectively a nearly natural three dimensional image to be viewed.Therefore, these systems are called multi-user or multiview systems.

One great problem, mainly if the number of perspectives is very high, isthat the hardware capacity of the visualization system used and, inparticular, the memory capacity required is very large.

The purpose of the invention presented here is to describe a method anddevice for the creation of three dimensional (especially moving) imageswith more than two perspectives (such as pseudo-holographic images) fromfed images with, in particular, only two perspectives as, for example,from left and right image channels, with which a relatively large numberof such perspectives can be synthesized with a relatively small amountof memory.

Furthermore, a device for the creation and display of three dimensional(mainly moving) images with more than two perspectives, especially inthe form of an autostereoscopic multi-viewer visualization system(multiview system), shall be described, with which, especially for acomparable large number of displayed perspectives, the requirements forthe hardware, mainly the memory, are relatively small and thereforerelatively inexpensive.

This problem will be solved with a method as described in claim 1 and adevice as described in claim 7 or with a multiuser visualization systemas described in claim 9.

One advantage of the invention described is that the number ofperspectives which have to be generated can arbitrarily be chosen by theuser.

The sub-claims contain useful extensions of the invention.

Additional details, features and advantages of the invention arecontained in the following description of exemplary and preferredembodiments.

The figures contain:

FIG. 1 a schematic diagram of the position and viewing angle ofdifferent viewers in several viewing zones of a multiview display;

FIG. 2 a schematic block diagram of a multiview system with a device forthe creation of images according to the invention;

FIG. 3 a schematic presentation of various correspondences, a right andleft occultation respectively, between a left image, a right image andsix in-between perspectives;

FIG. 4 a flow chart of a method according to the invention for thegeneration of three dimensional images;

FIG. 5 a schematic presentation of correspondences and a right and leftoccultation of FIG. 3 for a first synthesis situation;

FIG. 6 a schematic presentation of correspondences and a right and leftoccultation of FIG. 3 for a second synthesis situation;

FIG. 7 a schematic presentation of correspondences and a right and leftoccultation of FIG. 3 for a third synthesis situation;

FIG. 8 a schematic presentation of a matrix circuit for controllingevery lens of an adapter screen above a pixel or sub pixel of a display;and

FIG. 9 a schematic presentation of a display with regions for twodimensional or three dimensional visualization.

As previously mentioned, an autostereocopic multi-viewer visualizationdisplay DA (Multiview display) or autostereocopic screen is generatingaccording to FIG. 1 several viewing zones S1, . . . Sn, in which aviewer B1, . . . Bn can view an optimal three dimensional image.

For this purpose a number N (N>2) of perspectives of the image whichwill be presented, have to be merged (from now on called “merging”) sothat the autostereoscopic display based on the installed optic willsplit those perspectives for every viewer B1, . . . Bn, so that everyeye of every viewer B1, . . . Bn can view a different perspective inevery viewing zone S1, . . . , Sn and thus generates a three dimensionalimage impression for every viewer.

Consequently, in general a predefined number of in-between perspectiveshave to be created from two fed image perspectives, meaning a left and aright image, and to be merged so that the visualization system ordisplay can correctly process and display them.

The number and position of the viewing zones S1, . . . Sn, which will becreated depends on the number N of visualized and merged perspectives.The more perspectives which are available, the broader the viewing zoneswill be and the more the transition zones will be shifted to the leftand right border of the viewing area in front of the display DA.

If a viewer B1, . . . Bn moves in front of the display DA (or thescreen) from one viewing zone Sx to a neighboring viewing zone Sx+1, astripe of fuzziness will be seen to move through the image.

It is therefore the aim to have the number N of perspectives as large aspossible. But this will cause the problem that, with an increasingnumber of perspectives, the requirements for the hardware and especiallythe memory capacity of the underlying visualization system will increasesignificantly, which, in practice, will limit the number of generated ordisplayed perspectives.

With the invention presented here, a method and device for the creationof three dimensional images with a plurality of merged perspectives fromfed images, especially only two fed perspectives, will be describedwhich generates the displayed pixels with an arbitrary number ofin-between perspectives in such an efficient way that only those imagepixels of an in-between image (or in-between perspective) which actuallyhave to be displayed on the multiview display will be synthesized.

The image pixels will not be addressed separately, but will be createddirectly on the display, which means that the complete in-betweenperspectives never exist at any time and therefore do not need to bestored.

With the method according to the invention, the viewer has the option tochoose an arbitrary number N (N>2) of perspectives which have to bevisualized on the display or cinema screen. Furthermore, it is assumed,that only a number M of (fed) perspectives are stored in memory, so thata number N-M of in-between images or in-between perspectives have to besynthesized.

To simplify the remainder of the description, the commonly used case inwhich only two perspectives M, in particular a left and a right fedimage, are available in memory (or in a dedicated memory for the leftand right image). Of course, the method according to the invention canbe applied to more than two fed perspectives (M>2) for the creation ofthe required number N-M of in-between images or perspectives. In thiscase, the method will be used for the left-most and right-most fedperspectives, while the in-between fed perspectives will be used assupporting images.

The method according to the invention for the creation of the in-betweenimages (or in-between perspectives) covers a correspondence analysis andan in-between images (or in-between perspective) synthesis phase. Duringthe analysis phase for each pixel B(i,j) of the fed left image anattempt is made to find a corresponding right pixel B(i′,j′) (the terms“in-between images” and “in-between perspectives” will be usedsynonymously).

The index i or i′ defines the number of the row (row index) and theindex j or j′ defines the column (column index) within each image. Tosimplify the remainder of the description, it is assumed from now onthat i=i′ holds. This is feasible if the images are in normalized stereoform, which can always be achieved through a pre-processing ortransformation of the relevant images.

If the correspondence B(i,j)->B(i,j′) is known and correct, during thesynthesis phase all in-between images (in-between perspectives) can beextracted or synthesized by a virtual camera movement on the stereobasis from the left image to the right image (or vice versa).

If

j(α):=(1−α)*j+α*j′ for 0<=α<=1,

is set, it will be defined:

B(i,j(α)):=B(i,j) for 0<=α<=1.

This means, in particular, that

B(i,j(0))=B(i,j) and B(i,j(1))=B(i,j′)

holds, because B(i,j′) is the corresponding pixel of B(i,j).

If α moves from 0 to 1, then B(i,j) moves on a virtual camera path fromB(i,j) to B(i,j′), its corresponding pixel.

Provided that N-2 in-between images have to be created (meaning M=2), αksupporting points with

0<αk<1 for k=1, . . . , N−2,

have to be used, where the pixels B(i,j(αk)) will be combined into onein-between image. Here, preferably αk:=k/(N−1) will be definedequidistantly.

Now, right occultations are those pixels B(i,j), which only exist in theleft image and which are invisible for the right eye. For those pixelsthere is no correspondence in the right image. During a virtual cameramovement from the left image to the right image those pixels have to beremoved successively. This will start on the right border of the rightoccultation.

In a similar manner, left occultations are those pixels B(i′,j′), whichonly exist in the right image and which are invisible for the left eye.For those pixels there is no correspondence in the left image. During avirtual camera movement from the left image to the right image thosepixels have to be inserted successively. This will start on the rightborder of the left occultation.

FIG. 2 shows a schematic block diagram of a multiview system with adevice 100 according to the invention for the creation of threedimensional images with more than two perspectives. The device 100 isconnected via a first input with a left channel 1 for the feed with left(moving or still) images and via a second input with a right channel 2for the feed with right (moving or still) images, meaning twoperspectives of a three dimensional image. A controllableautostereoscopic multiview display 3 is connected via an output with thedevice 100.

The device 100 essentially includes a first image buffer 10 for at leastone left image and a second image buffer 11 for at least one rightimage, which are connected with the associated first or second input ofthe device 100. The images stored in buffer 10 and 11 will be fed to theanalysis unit 12 executing the correspondence analysis. The first outputof analysis unit 12 is connected with a first buffer 13 for a generatedindicator “Occultation(i,j)”, while the second output of the analysisunit 12 is connected with second buffer 14 for a pointer “Whereto(i,j)”as well generated by the analysis unit 12.

The indicator Occultation(i,j) stored in the first buffer 13 will be fedto a first input of an image synthesis unit 15, while the pointerWhereto(i,j) stored in buffer 14 will be fed to the second input of theimage synthesis unit 15. A third input of the image synthesis unit 15receives a left image from the first image buffer 10, while a fourthinput receives a right image from a second image buffer 11. The mergedleft and right images and the synthesized in-between images orin-between perspectives will be fed through the output of the imagesynthesis unit 15 to the autostereocopic multiview display 3 forpresentation.

The device 100 preferably also includes a buffer 31 for a matrix R(i,jr)of perspectives, which, for each subpixel of the connected display 3depending on its optical properties, describe which perspective (meaningfed (left or right) image or one of the synthesized in-betweenperspectives) has to be presented by that subpixel. The matrix R(i,j)has either to be computed before starting the display 3 with the methodaccording to the invention (FIG. 4A, 4B) and to be stored in buffer 31,or, if it is stored in the display, to be downloaded from the display 3to the buffer 31, or has to be fed in any other manner to the device100. The matrix can be computed in well-known ways for any applicableautostereoscopic display 3.

In the following description of a first embodiment according to theinvention each in-between image represents one of the in-betweenperspectives.

To improve the speed of the synthesis of the in-between images, unusedpixels of each in-between perspective will not be generated. Thefollowing procedure will be applied:

For each pixel/subpixel P(i,l) of the display 3, based on the previouslydescribed matrix, it can be identified from which perspective R(i,l)(0<=R(i,l <=N−1) the pixel to be presented has to be taken.

Therefore, it will be defined:

For all columns j of the image area and each k=1, . . . , N for whichthere is a correspondence in the right image, 1:=(1−αk)*j+αk*j′ computeand set P(i,l):=B(i,j), if k=R(i,l) is true.

For the left and right occultations, for which there is nocorrespondence B(i,j)->B(i,j′), the procedure is as follows:

A right occultation, which is present in the left image and not in theright will be faded out during the virtual camera movement from left toright. In this case, the occultation will move from the rightoccultation border to the left. The length of the visible occultationstripe in the row is known for each perspective. Furthermore, it isknown where this stripe in perspective k will be mapped to. Therefore, acheck can be made for all pixels whether k=R(i,l) is true. Only thosepixels will be inserted at P(i,l).

This will be executed in the same way for the left occultations.

In a second embodiment according to the invention, again each in-betweenimage will represent one perspective.

In this case, it will be computed backwards which perspective and whichdisplacement has to be applied for each display pixel/display subpixel.

In detail, the procedure will be as follows:

Again it is assumed that the images are available in stereo normalizedform, such that each row in the left image corresponds with itsassociated row in the right image. This again can be achieved by apre-executed transformation.

Furthermore, the columns i shall be fixed.

For each pixel B(i,j) in the left image an indicator Occultation(i,j)and a pointer Whereto(i,j) will be generated with the analysis unit 12.

The indicator Occultation(i,j) shows whether there is a correspondingpixel B(i,j′) in the right image for the pixel B(i,j) of the left image,if it is contained in a right occultation or if there is a leftoccultation in the right image left of this pixel. Based on this thevalues of Occultation(i,j) will be defined by the analysis unit 12 asfollows:

-   -   1, if there is a left occultation left of the pointer        Whereto(i,j),

Occultation(i,j)=0, if there is a correspondence in the right image,

-   -   −1, if the pixel B(i,j) is part of a right occultation.

For the pointer Whereto(i,j) it is Whereto(i,j)=j′, ifOccultation(i,j)=0 is true, otherwise Whereto(i,j) is undefined.

In FIG. 3, the values of Occultation(i,j) defined above as well as thedifferent possibilities of a correspondence of the right occultations RVand a left occultation LV are shown for eight perspectives, namely aleft image LB, first to sixth in-between images ZB1, . . . , ZB6 and aright image RB.

This method according to the invention will be applied to each row i ofthe image according to the flow chart described in FIGS. 4A and 4B.

After starting the method, in a step S1 a pointer jr to a subpixel and apointer j1 to a pixel will be set to 0. Furthermore, the pointer j1 willbe incremented in a step S1 a by 1 until the indicator Occultation(i,j)is larger or equal to 0 (a check for this is implied as part of step S1a).

When this is the case, a loop starts to run with step S2 through allsubpixels jr of row i (to which the method is currently applied).

If the check in step S2 indicates jr is larger than the number ofsubpixels per row, the method will stop for the current row i with step7 and, if necessary, will be started with “Start” for the next row.Otherwise, according to step S3 for jr the next perspective R(i,jr),which has to be displayed for subpixel jr will be fetched from buffer31.

If the check with step S4 indicates perspective “R(i,j)=0” (“yes”), thesubpixel will be fetched according to step S8 from the left image.Additionally, according to step S10, jr will be incremented by 1 and itwill then be stepped back to step S2 to repeat this procedure.

But if the check returns “no” in step S4 (meaning R(i,jr) is not 0), instep S5 it will be tested if “R(i,jr)=N−1” (N=number of perspectives) istrue. If this is the case, the subpixel will be fetched according tostep S9 from the right image. Additionally, according to step S10, jrwill be incremented by 1 and it will then be stepped back to step S2 torepeat this procedure.

But if the step S5 returns “no” (meaning R(i,jr) not equal N−1), thecore search process will start with step S6 with defining “above=false”and “below =false”.

If the question of step S11 either “above=false” and “below =false” isanswered with “yes”, then the subpixel is not yet encapsulated fromabove or below and the search process has to be continued with steps S12to S17. But if step S11 can be answered with “no”, the subpixel isencapsulated and can be fetched according to step S18 either from theleft or right image. For this step four cases have to be distinguished:

Case 1: j_above−j_below <=1 andWhereto(i,j_above)−Whereto(i,j_below)<=1:

In this case there is a unique correspondence and the subpixel can befetched from the left image. This case is schematically shown in FIG. 5.Here, a pixel from the left image has to be selected, which is theresult from j_above and j_below in in-between image (in-betweenperspective) ZB2.

Case 2: j_above−j_below >1 and Whereto(i,j_above)−Whereto(i,j_below)<=1:

In this case there is a right occultation and the subpixel can befetched from the left image. This case is schematically shown in FIG. 6.

Case 3: j_above−j_below <=1 and Whereto(i,j_above)−Whereto(i,j_below)>1:

In this case there is a left occultation and the subpixel can be fetchedfrom the right image. This case is schematically shown in FIG. 7.

Case 4: j_above−j_below >1 and Whereto(i,j_above)−Whereto(i,j_below)>1:

This case is not feasible in reality, as a right and a left occultationcannot be in the same area at the same time.

After that, step S18 according to FIG. 4A and step S10 jr will beincremented by 1, stepped back to step S2 and the process repeated.

But if the question according to step S11 is answered with “yes”,according to step S12 a new positition “j1_PositionNew” will becomputed. Furthermore, depending on perspective R(i,jr) it will becomputed where the pixel B(i,j1) would be mapped to.

If, according to the check in step S13, the new position isj1_PositionNew=Pixel(i,jr), then the mapped pixel is found. Then,according to step S15, “above=true” and “below=true”, “j_above=j1” and“j_below =j1” will be set and the subpixel will be fetched by answeringthe question of step S11 with “no” according to step S18 and the casesdescribed above.

But if the check in step S13 returns “no” and according to the questionof step S14 the new position is “j1_PositionNew>pixel(i,jr)” (“yes”),this pixel is located above pixel(i,jr). Now, according to step S16,“above=true” and “j_above=j1” will be set and j1 decremented by 1.Furthermore, it will be checked (in the flow chart part of step S16),whether the indicator Occultation(i,j) is larger or equal to 0. If thisis not the case, j1 will be decremented until it is the case.

If the check in step S14 returns “no”, the new position is belowpixel(i,jr). Now according to step S17 “below =true” and “j_below =j1”will be set and j1 incremented by 1. Furthermore, it will be checked (inthe flow chart part of step S17), if the indicator Occultation(i,j) islarger or equal to 0. If this is not the case, j1 will be incrementeduntil it is the case.

After steps S15, S16 and S17 respectively, a backstep to step S11 isexecuted and the loop will be continued with a new check, whether it is“above=false” or “below=false”.

Modifications of the above described method can be found by applying adifferent optimization strategy, for example if the pixel j1 is not tobe taken from the previous subpixel jr, the pixel j1 will be takeninstead from the previous row. In some cases, this can eliminate a fewsteps in finding the optimal pixel.

For example, it would be possible to merge the indicator Occultation andthe pointer Whereto into one array. In case of a right occultation, itcould be set Whereto(i,j):=−1. This indicates that there is nocorrespondence. A left occultation can be identified whenWhereto(i,j_above)-Whereto(i,j_below)>1.

With a method according to the invention, a large number N ofperspectives can be set, for example N=1024 or 2048, such that therewill be completely new potentials for the 3D visualization of images.

This revealed that with an increasing number N of perspectives theimages become clearer and the transition zones between the viewing zoneswill become smaller and smaller.

It could be thought that the image quality will be reduced, becausefewer pixels will be available for each individual perspective. But thisis not the case, because, for each eye of a viewer, the number ofvisible pixels depends on the size of the subpixel and the size andfocal length of the lenticular lenses or parallax barriers. Moreperspectives are visible there. But these are more consistent, becausetheir distance is smaller on the stereo basis.

A pseudo-holographic effect results, if the stereo basis between theleft and right image is significantly large (for example 80-90 cm). Inthis case the viewer moves through the different viewing zones for alonger time and has a greater impression of “moving around” the object.

The methods described here are applicable for autostereoscopic displaysas well as for autostereocopic adapter screens based on lenses orparallax barriers.

Furthermore, it is possible to execute the step of a correspondenceanalysis, not only where the images are presented, but also where theimages are captured and to transmit it together with the stereo imagesusing a data compression method (for example MPEG-4) to the locationwhere it will be displayed.

Additionally, it is possible that the correspondence analysis will begenerated by a transformation of a transmitted depth map.

Preferably, the method according to the invention can be executed by acomputer program.

The implementation of the methods according to the invention is possiblefor example according to FIG. 8 with a plurality of lenses L (especiallynano lenses in liquid lens technology), placed in front of a pixel (RGBcolor) or subpixel (only R or G or B-color) of a display, where thelenses L will be controlled by a horizontal and vertical electronicmatrix circuit VM, such that only those pixels of an underlying displaywhich are needed for the visualization of the corresponding synthesizedin-between images will be activated or created.

Because every pixel is individually controllable, certain areas of thedisplay DA can be switched according to FIG. 9 into 3D mode, while otherareas are still in 2D mode. For this purpose, the on and off impulsewill be fed row and column sequentially. Only those lenses which receivethe horizontal and vertical impulse at the same time will be switched onor off.

1. A method for the creation of three-dimensional images forvisualization on an autostereoscopic display or an autostereoscopicscreen, comprising: creating a number N>2 of perspectives, from fedimages with a number M of perspectives, where M<N; and creating N−Min-between perspectives in such a way that the pixels which do not haveto be presented on the display or screen for visualization of thein-between perspectives are not synthesized.
 2. The method according toclaim 1, wherein the pixels of an in-between perspective are notaddressed separately, but are created directly on the display.
 3. Themethod according to claim 1, wherein: the number of fed images, M=2; andthe fed images are perspectives in the form of a left and right image.4. The method according to claim 1, further comprising, in an analysisphase for each pixel B(i,j) of a fed image of a first perspective,searching a corresponding pixel B(i′,j′) of a fed image a second.
 5. Themethod according to claim 4, further comprising, during a synthesisphase, synthesizing N−M in-between perspectives synthesized through avirtual camera movement along the stereo basis between a pixel B(i,j) ofa fed image of a first perspective and the corresponding pixel B(i′,j′)of a fed image of a second perspective.
 6. The method according to claim5, further comprising choosing for which, for the creation of N−2 (M=2)in-between perspectives, αk supporting points with 0<αk<1 for k=N-2,wherein will be chosen, where the pixels B(i,j(αk)) are to be combinedto an in-between perspective.
 7. A device for the execution of a methodfor the creation of three-dimensional images for visualization on anautostereoscopic display or an autostereoscopic screen, comprising: ananalysis unit for the correspondence analysis between pixels B(I,j) of afed image of a first perspective and pixels B(i′,j′) of a fed image of asecond perspective; and an image synthesis unit for the synthesis ofpixels of at least one in-between perspective that have to be visualizedon the display or screen, wherein pixels of an in-between perspectivewhich do not have to be visualized on the display or screen are notsynthesized.
 8. The device according to claim 7, wherein: the analysisunit generates, for each pixel B(i,j) of a fed image of a firstperspective, an indicator Occultation(i,j) and a pointer Whereto(i,j);the indicator Occultation(i,j) indicates whether there is acorresponding pixel B(i,j′) in the fed image of a second perspective forthe pixel B(i,j) in the fed image of a first perspective, whether it ispart of a right occultation or whether there is a left occultation infront of the pixel B(i,j) in the image of a second perspective; and, thepointer Whereto(i,j)=j′ holds, if the indicator is Occultation(i,j)=0.9. A device for the visualization of three dimensional, especiallypseudo-holographic images, especially in form of an autostereoscopicmulti-user visualization system, comprising: an analysis unit for thecorrespondence analysis between pixels B(i,j) of a fed image of a firstperspective and pixels B(i′,j′) of a fed image of a second perspective;an image synthesis unit for the synthesis pixels of at least onein-between perspective that have to be visualized on the display orscreen, wherein pixels of an in-between perspective which do not have tobe visualized on the display or screen are not be synthesized; and anautostereoscopic display or an autostereoscopic screen.
 10. The methodaccording to claim 4, wherein: the first perspective is a leftperspective; and the second perspective is a right perspective.